In an apparent move to avoid stoking a fresh controversy as the Bharatiya Janata Party comes to terms with a bitter separation from the Janata Dal-United, Gujarat Chief Minister Narendra Modi has declined an invitation to visit Ayodhya, his office said on Tuesday.